feat(core-ts): entry_refs and reference integrity, gated off (RFC-0045 step 2a, #114) - #205

RFC-0045 rollout step 2a ("Store and integrity", first half) for #114. The PM approved splitting step 2 in two.

• Production keeps refusing reference and repeater on every path (Studio, MCP, API, import, sync, push, pull, restore). No production caller passes a FieldTypeSupport.
• The shared suites enable the types with SiteGraphStoreOptions.fieldTypes and CopySiteOptions.fieldTypes.
• No consumer is half-wired: everything listed under "Moves to 2b" is absent from this branch, not stubbed.
• main stays releasable. The only behaviour changes for existing sites are the phase-2 ones the owner accepted (decision 3), and the changeset lists them.

This PR first held all of step 2 in ten commits. It was force-pushed to the 2a scope; the full version is kept on a local branch for 2b.

What 2a contains, per §9 step 2 bullet

• Migration 0033 and entry_refs in SITE_TABLES.

  • packages/storage/migrations/0033_entry_refs.ts: the table and the entry_refs_target index, DDL only.
  • migrate.test.ts lists it.

• The target guard, the definitions anchor bound as planned, and rows with shared/root_key.

  • SQL fragments are in core-ts/src/references/sql.ts: membership, "shown by a published entry", the target guard, and usableMembers.
  • Both anchors read the fragment's stored text when the write is planned and check it against the planned Site; a mismatch is a ConflictError. The batch then compares that exact text, with SQL NULL when the fragment is absent.
  • The writer is references/writer.ts. It turns member ids into group ids in changed values. Before the batch it checks unknown_entry, reference_target_moved, the entry's own group and where.
  • The target guard covers changed keys, plus every own value when the entry goes live.
  • The definitions anchor applies to every write in a site with reference fields, and elsewhere to writes that change fields or collection.
  • Rows: an anchored DELETE, then a plain INSERT … SELECT that reads shared from the site row at batch time, by key or fromKey, exact key first. The read is an aggregate, which workerd resolves, and only reference or repeater definitions match.
  • A NULL shared fails the batch. Only NOT NULL constraint failed: entry_refs.shared maps to ConflictError.
  • store/entry-integrity.ts plans the links, references and leave rules together.

• Delete, move, group-change and publish rules, with the collections anchor and the policy.

  • references/leave.ts: every delete, move and group change carries the collections anchor.
  • The guard is NOT EXISTS(blocking) OR (usable members AND NOT EXISTS(blocking with a pair not in P)). A move keeps the pairs whose field allows its new collection.
  • Usable members are one json_each count over the needed collection lists. It binds six parameters however many pairs block.
  • Refusals: entry_referenced (up to 20 referrers and the total) and reference_target_moved.
  • Publishing re-checks every own value.

• R1 to R16 as shared-suite cases. See the table below.

• Chunked entryRefsRebuild in import, sync and restore (references/rebuild.ts).

  • Emitted in nodeBatch under the same condition as refsRebuild.
  • Chunked below 90 KB.
  • shared is computed in JS; keys are read through fromKey.

• The membership rule everywhere, the relink refusal, the written-group guard, and the readsOriginal/originalOf tightening.

  • checkTranslationLinks returns the written-group guard (omitted for the node's own id) and the dependants guard (translation_group_in_use). Pages get them too.
  • A group-only change gets the source-group check.
  • The tightening is in fields/resolve.ts and store/entry-fields.ts.

• field_operation_running. A marker that converts values into references refuses deletes, moves and group changes in its target collections. The collections anchor fails a batch that was planned before the marker landed.

• Canonical bodies in importSite, and CopySiteOptions.fieldTypes.

  • nodeBatch stores canonical JSON.
  • The capability reaches checkBundle and the snapshot gate.
  • The §2.4 canonical-rows job is not needed: the anchors compare stored text, so key order never matters. It was in d107895 and was removed after review.

• "Operations naming the new types refused" (half of that bullet). checkOperation answers field_operation_unsupported. Phases A and C check definitions with the store's capability.

• §2.8 definition changes (references/definition-changes.ts via store/site-fields.ts), needed by R3:

  • field_key_in_use, for a new field or a new reference sub-field over keys that items hold;
  • a removed target collection whose values still have members there;
  • a removed or retyped sub-field that holds values;
  • a change of multiple.

  Each count is guarded on the site CAS.

• Refusal codes. 409 or 422 in api/src/errors.ts, and MCP hints in field-hints.ts. translation_group_in_use can already happen in production.

  • A referrer write naming an entry outside the field's collections is unknown_entry (422). reference_target_moved (409) is only for moves.

• List rows after the tightening. The list-values fingerprint format moves to 2, so listed collections rebuild once.

• Step-1 review note (held blobs). prepareImport runs the gate on site bodies that the bundle's new snapshots point at but the target already holds (not for a capable site). The test fails without the fix.

Moves to 2b

2b branches from main after this merges. These are exactly the step-2 bullets, or parts of bullets, that are not here:

1. referenceIssues in cloudhal import --dry-run (the second half of the rebuild bullet), with OQ 14's relinked-original warning.
2. The item walk in fieldAssetRefs, fieldAssetIssues with sub-field types and item paths, and the prefix-match skip set (entry-fields.ts:107, field-refs.ts:81).
3. Rows and the target guard in field-operation rewrites (the first half of that bullet; the refusal is in 2a).
4. The kind-aware release depsOf (core-ts/src/release-deps.ts).
5. Edition id mapping in forEdition, with emptied items dropped.
6. The fields: 2 capability in the gate (siteFieldTypeSupport, FIELDS_FEATURE_VERSION, FIELDS_SUPPORTED).
7. The API: entries/pick and entries/:id/references, and their scopes. The entry routes' new status codes are already in 2a.

None of these is reachable in production before step 7, because no reference value can exist.

R-cases and where they are tested

The cases are in packages/core-ts/src/testing-references.ts, which runs on every backend through describeSiteGraphStore. Each one is two interleaved batches via racingDriver.

The suite also covers:

• writes and canonical ids;
• refused targets: missing, cross-site, wrong collection, own group, archived;
• a dangling value at publish;
• published versus draft referrers, and archiving does not release the lock;
• other members, moves and group changes;
• relink, then publish (refused), then delete;
• translation_group_in_use;
• a translation becoming its own original;
• the NULL anchor;
• the shared rule;
• a foreign tenant's row;
• canonical import bodies and the held-blob gate;
• a verbatim (non-canonical) site row taking writes, deletes and relinks;
• 22 blocking pairs on a delete and a group change;
• a reference sub-field added over a key that items hold;
• the operation refusal.

test/references-store.test.ts (sqlite) covers:

• the anchors for every fixtures/valid/site file, stored canonically and with every key order reversed, including fractional and 1e21 bounds; a mismatch with the planned site, and a change after planning;
• the shared subquery: fromKey, the exact key winning, and a retyped or missing key failing;
• rows through fromKey;
• rebuild chunks below 90 KB;
• the NOT NULL mapping (an unrelated column is not mapped);
• (the canonical-rows job test is gone with the job).

Deviations

1. Usable members are checked per pair, not through the intersection A. Each blocking field needs another usable member in its own collections. That is the stated invariant, and it does not over-refuse when two fields allow different collections.
2. Dropping translation from an original whose group stays its own id is not refused. The relink refusal applies when a node's group changes.
3. Member ids are turned into group ids only in values the write changes. Doing it to unchanged ids would contradict the RFC's "relink, then publish the referrer (refused)" case.
4. The anchors do not use the RFC's json(…) IS json(canonical). They compare the fragment's stored text, read when the write is planned and checked against the planned Site. There is no second read the plan did not see, and key order never matters, so the §2.4 canonical-rows job is not shipped. (PM review, A.)
5. entry_referenced.details.referrers lists the holders of blocking rows, not the translations that inherit from them.
6. A reference to the entry's own group, and a where miss on a newly chosen target, are refused as invalid_value (invalid_fields). The RFC names no code for either. where is checked before the batch only, as the RFC says.
7. set_field_localized phase A's UPDATE entry_refs SET shared is not wired, because operations on reference fields are refused until step 4.
   • The definitions anchor now covers every write in a site with reference fields (PM review, B). The RFC binds it only when values change.
   • So a write racing any phase A fails and retries; the batch-time shared read is a second line of defence. R16 simulates phase A with a raw write of the site row.
8. The §2.8 refusals and field_key_in_use are added, including a reference sub-field added over keys that items hold. They are not in the literal bullet list, but R3 needs them and the step-1 hand-over assigned them to step 2.

Tests run (locally, one file at a time)

Run on cb0ad73:

• The new suite on all three backends, each 39/39: test/store.test.ts -t "<backend>.*RFC-0045 step 2" for better-sqlite3, D1 (Miniflare) and Durable Object (Miniflare).
• test/store.test.ts -t better-sqlite3: 204 passed.
• Other core-ts files: references-store 22, list-values 51, list-values-rows 7, list-values-sql 7, list-values-query 18, sync 10, sync-plan 9, sync-restore 5, multi-site 6, references-core 22, fields-resolve 16.
• cloudflare: list-values-job and scheduler (11).
• mcp-server: tool-schemas 6.
• Typecheck: core-ts, cloudflare, api, mcp-server, cli and astro-renderer.
• pnpm size:check passes; biome reports no errors.

Not run locally: the D1 and Durable Object variants of the rest of store.test.ts (left to CI), and the full suites of the other packages.

🤖 Generated with Claude Code